{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train and Host a Keras Sequential Model\n",
    "\n",
    "This notebook shows how to train and host a Keras Sequential model on SageMaker. The model used for this notebook is a simple deep CNN that was extracted from [the Keras examples](https://github.com/keras-team/keras/blob/master/examples/cifar10_cnn.py)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The dataset\n",
    "The [CIFAR-10 dataset](https://www.cs.toronto.edu/~kriz/cifar.html) is one of the most popular machine learning datasets. It consists of 60,000 32x32 images belonging to 10 different classes (6,000 images per class). Here are the classes in the dataset, as well as 10 random images from each:\n",
    "\n",
    "![cifar10](https://maet3608.github.io/nuts-ml/_images/cifar10.png)\n",
    "\n",
    "In this tutorial, we will train a deep CNN to recognize these images."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set up the environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import sagemaker\n",
    "from sagemaker import get_execution_role\n",
    "\n",
    "sagemaker_session = sagemaker.Session()\n",
    "\n",
    "role = get_execution_role()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download the CIFAR-10 dataset\n",
    "Downloading the test and training data will take around 5 minutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "import utils\n",
    "\n",
    "utils.cifar10_download()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Upload the dataset to an S3 bucket"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs = sagemaker_session.upload_data(path='/tmp/cifar10_data', key_prefix='data/DEMO-cifar10')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`sagemaker_session.upload_data` will upload the CIFAR-10 dataset from this machine to a bucket named **sagemaker-{region}-{*your aws account number*}**, if you don't have this bucket yet, `sagemaker_session` will create it for you."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Complete source code\n",
    "Here is the full source code for the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!cat cifar10_cnn.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets take a closer look:\n",
    "\n",
    "### The model function\n",
    "This function constitutes the main difference between TensorFlow and Keras models on SageMaker; Keras models have a `keras_model_fn`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def keras_model_fn(hyperparameters):\n",
    "    \"\"\"keras_model_fn receives hyperparameters from the training job and returns a compiled keras model.\n",
    "    The model will be transformed into a TensorFlow Estimator before training and it will be saved in a \n",
    "    TensorFlow Serving SavedModel at the end of training.\n",
    "\n",
    "    Args:\n",
    "        hyperparameters: The hyperparameters passed to the SageMaker TrainingJob that runs your TensorFlow \n",
    "                         training script.\n",
    "    Returns: A compiled Keras model\n",
    "    \"\"\"\n",
    "    model = Sequential()\n",
    "\n",
    "    model.add(Conv2D(32, (3, 3), padding='same', name='inputs', input_shape=(HEIGHT, WIDTH, DEPTH)))\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(Conv2D(32, (3, 3)))\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "    model.add(Dropout(0.25))\n",
    "\n",
    "    model.add(Conv2D(64, (3, 3), padding='same'))\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(Conv2D(64, (3, 3)))\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "    model.add(Dropout(0.25))\n",
    "\n",
    "    model.add(Flatten())\n",
    "    model.add(Dense(512))\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(Dropout(0.5))\n",
    "    model.add(Dense(NUM_CLASSES))\n",
    "    model.add(Activation('softmax'))\n",
    "    \n",
    "    opt = RMSPropOptimizer(learning_rate=hyperparameters['learning_rate'], decay=hyperparameters['decay'])\n",
    "\n",
    "    model.compile(loss='categorical_crossentropy',\n",
    "                  optimizer=opt,\n",
    "                  metrics=['accuracy'])\n",
    "\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This function builds and returns a compiled Keras model.\n",
    "\n",
    "**Note:** The first layer is named `PREDICT_INPUTS`. This serves as a workaround for a known issue where TensorFlow does not recognize the default (or any custom) name for the first layer of Keras models. Furthermore, note that we are wrapping our model in a `tf.keras.Model` before returning it. This serves as a workaround for a known issue where a Sequential model cannot be directly converted into an Estimator. See [here](https://github.com/tensorflow/tensorflow/issues/20552) for more information about the issue.\n",
    "\n",
    "### Input functions\n",
    "These functions are similar to those required by any other model using the TensorFlow Estimator API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def serving_input_fn(params):\n",
    "    # Notice that the input placeholder has the same input shape as the Keras model input\n",
    "    tensor = tf.placeholder(tf.float32, shape=[None, HEIGHT, WIDTH, DEPTH])\n",
    "    \n",
    "    # The inputs key INPUT_TENSOR_NAME matches the Keras InputLayer name\n",
    "    inputs = {INPUT_TENSOR_NAME: tensor}\n",
    "    return tf.estimator.export.ServingInputReceiver(inputs, inputs)\n",
    "\n",
    "\n",
    "def train_input_fn(training_dir, params):\n",
    "    return _input(tf.estimator.ModeKeys.TRAIN,\n",
    "                    batch_size=BATCH_SIZE, data_dir=training_dir)\n",
    "\n",
    "\n",
    "def eval_input_fn(training_dir, params):\n",
    "    return _input(tf.estimator.ModeKeys.EVAL,\n",
    "                    batch_size=BATCH_SIZE, data_dir=training_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `train_` and `eval_` functions call the `_input` function which returns a properly processed and shuffled (for training) set of images and labels.\n",
    "\n",
    "## Create a training job using the SageMaker TensorFlow Estimator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from sagemaker.tensorflow import TensorFlow\n",
    "\n",
    "estimator = TensorFlow(entry_point='cifar10_cnn.py',\n",
    "                       role=role,\n",
    "                       framework_version='1.12.0',\n",
    "                       hyperparameters={'learning_rate': 1e-4, 'decay':1e-6},\n",
    "                       training_steps=1000, evaluation_steps=100,\n",
    "                       train_instance_count=1, train_instance_type='ml.c4.xlarge')\n",
    "\n",
    "estimator.fit(inputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: Keras models have a known issue and cannot be used for distributed (multi-instance) training. Keep `train_instance_count == 1` until the TensorFlow/Keras team support this feature. See [here](https://github.com/tensorflow/tensorflow/issues/14504) for more information about the issue.\n",
    "\n",
    "\n",
    "## Deploy the trained model\n",
    "\n",
    "The deploy() method creates an endpoint which serves prediction requests in real-time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictor = estimator.deploy(initial_instance_count=1, instance_type='ml.m4.xlarge')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Make some predictions\n",
    "Prediction is not the focus of this notebook, so to verify the endpoint's functionality, we'll simply generate random data in the correct shape and make a prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating fake prediction data\n",
    "import numpy as np\n",
    "data = np.random.randn(1, 32, 32, 3)\n",
    "\n",
    "# The inputs key 'inputs_input' matches the Keras InputLayer name\n",
    "predictor.predict({'inputs_input': data}) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cleaning up\n",
    "To avoid incurring charges to your AWS account for the resources used in this tutorial you need to delete the SageMaker Endpoint:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sagemaker.Session().delete_endpoint(predictor.endpoint)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_tensorflow_p36",
   "language": "python",
   "name": "conda_tensorflow_p36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  },
  "notice": "Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.  Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
